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(54) Use of proxy servers to provide annotation overiays 

(57) A system and method for providing annotation 
overlays from diverse sources of commentary for World- 
Wide Web documents is disclosed. Sources of com- 
mentary contribute annotation overlays regarding par- 
ticular documents on the World-Wide Web. The 
annotation overlays from a particular source are stored 
on one or more overlay servers, which are connected to . 
the Web. A user of a Web browser opens an annotation 
proxy server between the Web browser and the Web 
senders that intercepts all documents retrieved by the 
Web browser and merges with the retrieved documents 
commentary from sources designated by the user of the 
Web browser that refer to the requested documents. 
Multiple annotation overlay proxies can be serially con- 
nected. The annotation proxy can perform the merge 
operation by first creating a local annotation directory of 
annotation overlays from sources designated by the 
user tiien, when the user requests a document, merging 
with the requested document information only from the 
annotation directory. Alternatively, the annotation proxy 
can perfbnn the merge operation on the fly by pulling 
the annotation overlays directiy from the Web servers 
without the use of a local annotation directory 
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Description 

The present invention relates generally to computer networks, and particularly to proxy servers used to provide 
annotation overlays for documents stored on computer networks. 

5 

BACKGROUND OF THE INVENTION 

The World-Wide Web (n/VWW") links many of the servers making up the Internet, each storing documents identi- 
fied by unique universal resource locators (URLs). Many of the documents stored on Web servers are written in a stand- 

10 ' ard document description language called HTML (hypertext markup language). Using HTML, a designer of Web 
documents can associate hypertext links or annotations with specific words or phrases in a document (these hypertext 
links identify the URLs of other Web documents or other parts of the same document providing information related to 
the words or phrases ) and specify visual aspects and the content of a Web page, 

A user accesses documents stored on the WWW using a Web browser (a computer program designed to display 

15 HTML documents and communicate with Web servers) running on a Web cfient connected to the Internet Typically, this 
is done by the user selecting a hypertext link (typically displayed by the Web browser as a highlighted word or phrase) 
within a document being viewed with the Web browser. The Web browser then issues a HTTP (hypertext transfer pro- 
tocol) request for the requested document to tfie Web server identified by the requested document's URL In response, 
the designated Web server returns the requested document to the Web browser, also using the HTTP 

20 _ Many entities, espedally corporations ttial allow access from corporate systems to the Web. nxxiify this document 
access, process by providing a firewall proxy running on a proxy server situated between the Web client running the 
browser and ttie various Web servers hosting the requested documents. In this modified situation, all HTTP requests 
issued by the browser and all documents returned by the Web senders simply routed through ttie firewall proxy, which 
implements a proxy server communications protocol that is a subset of the HTTP. Apart from providing a buffer between 

25 the Web dient and servers, a pure firewall proxy performs no additional operations on the transferred infomnation. 
Another common type of firewall proxy is a caching firewall proxy, which caches requested documents to provide faster 
sut^sequent access to those documents. 

The ease of access and page design provided by the Web has proved attractive to many types of uses; e.g., indi- 
viduals and corporations, who have not traditionally used the Internet Additionally, the WWW is increasingly being used 

30 for commercial purposes, such as advertising and sales. Together, the new users and uses mean that an information 
explosion is occurring on ttie Web. With this information explosion it is becoming increasingly important tinat Web users 
be able to comment on the content of Web document view the commentary of others, or filter Information in Web 
pages. For example, a competitor or industry aitic might wish to comment on product announcements made by another 
competitor, buyers of a specific sennce might want to access the commentary of certain aitics (but not otiiers) regard- 

35 ing ttiat service and parents might wish to block their children's access to aO documents classified as inappropriate by 
a review board with whose opinions they agree. Ideally, these features would be implemented in a manner that is com- 
patible witii existing Web browsers and the HTTP. 

One system that provides a subset of these features by taking advantage of the proxy sender protocol is tiie Open 
Software Foundation's Worid Wide Web Agent Toolkit or OreO. OreO allows users to build personal agents ttiat can 

40 perform f fltering functions on requested documents before they are viewed using tiie Web browser. The agents created 
with OreO can be used in pipeline anywhere between a traditional Web client (i e.. Web browser) and a Web server to 
perform more complex and varied filtering of Web transactions. For example, a user could connect an obscenity filter in 
series witii a violence filter to ensure appropriate Web browsing for their children, OreO makes this pipelining possible 
by providing agent interfaces that make each agent took like a traditional Web client on one side and a proxy sender on 

45 the ottier. 

However, because the OreO toolkit does not address tfie creation of source libraries of commentary associated 
with known commentators and critics. OreO agents are not well-suited to merge commentary by sources other ttian ttie 
creator of a requested document with the requested document Moreover. OreO agents can only perform filtering by 
parsing all requested documents looking for occurrences of certain key phrases or patterns then deleting or replacing 

so those key phrases or patterns. Cleariy then, unless a commentator creates a new agent for each new document or 
class of documents and makes those agents available to all interested Web users, which would be extremely unwieldy, 
many alternate words and phrases equivalent to the key words and phrases would be missed. Finally, because the 
agents are not Web documents, it would not be possible to provide overiays to comment on tiie changes made by a first 
agent witiiout anottier commentator aeating a second agent and making that second agent available for users to insert 

55 between tiie Web browser and the first agent 

Therefore, tiiere is a need for a system ttiat introduces a proxy sender between Web servers and clients that allows 
parts of requested documents to be annotated, filtered, transformed or deleted before ttie documents are viewed witii 
a Web browser. Unlike ttie OreO agent ttiis system should perform ttie aforementioned annotating, filtering, transform- 
ing and deleting based on sources of commentary associated witti Web servers ttiat might be completely unrelated to 
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the author of the requested document. Ideally, a user should be able to indicate to the proxy server specific overlay 
sources to merge. Then, when the user requests a document, that request should be relayed through the proxy, which 
merges the requested document with overlays from the user-specified sources that reference the requested document. 
The resulting merged document should be viewable with any existing Web browser. 
5 Alternatively, the system should allow a user of the proxy to direct the proxy to form a library of annotations from a 
specific set of sources. Then, when a user requests a document the proxy should be able to merge comments in the 
library of annotations with the requested document, eliminating the need to search the Web for the appropriate anno- 
tations. Ideally, each of the overlays should have their own URLs so they could be easily annotated by other commen- 
tators. 

'10 

SUMMARY QF THE INVENTION 

In summary, the present invention is a system and method for merging annotations from various sources with doc- 
uments requested over the Web in such a way that the merged document is displayable by existing Web browsers. 

IS Specifically, the present invention Is a system for providing annotation overlays for documents requested over a 
computer network that incorporates a plurality of servers to store the documents. Each stored document has a unique 
document identifier and is viewable from a client computer having a browser corrfigured to request and receive docu- 
ments over the network. Features of the present invention include at least one stored overiay group associated with one 
of the servers. Each such overlay group encapsulates annotation overlays regarding at least one document and has a 
JO unique source identifier. Another feature of the present Invention is an annotation oveday proxy (AOP). which is a soft- 
ware routine configured to merge a requested document from a first s^er with associated annotation overlays regard- 
ing the requested document from specified overiay groups. The annotation overiay proxy then relays the merged 
document to a recever unit that is selected from another proxy (possibly a firewall proxy or another annotatton overiay 
proxy) or the.browser, which ultimately displays the merged document 

25 The present invention is also a method usable in the same type of computer network for providing annotation over- 
lays for a requested document As a first step, at least one stored overiay group is associated with a network document 
server. A merged document is then formed by merging a requested document stored on a first server with annotation 
overlays regarding the requested document from specific overiay groups. This merged document Is then relayed to a 
receiver selected from another proxy or said browser. 

30 

BRIEF DESCRIPTION OF THE DRAWINGS 

Additional objects and features of the invention will be more readily apparent from tiie following detailed description 
and appended claims when taken in conjunction with the drawings, in which: 

35 

Rgure 1 is a block diagram of a preferred embodiment of the present invention. 

Figure 2 is a block diagram of the preferred embodiment showing the situation where two annotation overiay prox- 
ies are connected in series. 

40 

Figure 3 is a flow chart of a prefen-ed method for merging annotation overiays and requested documents. 

Figure 4 is a diagram illustrating how the annotation overiay proxy merges annotations with a requested document. 

45 Rgure 5 is an illustration of the appearance of a merged document dsplayed by a web browser. 

Rgure 6 is a block diagram of an alternative embodiment of tiie present invention where the annotation overiay 
proxy does not build or use the annotatk)n directory of Figure 1 . 

so Figure 7 is a block diagram of an alternative embodiment of the present invention where each annotation overiay 
can refer to one or more documents. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

55 Referring to Rgure 1 . there is shown a block diagram of a preferred embodiment 100. which includes a Web client 
1 10. a proxy server 130 coupled to the Web client 1 10 and Web servers 140. 142. 144, each of which are coupled to 
the proxy server 130. 

A Web browser 1 12 executes In the Web client 1 10. while an annotation overiay proxy (AOP) 1 1 4 executes in the 
proxy sen/er 130. Infbnnation output by the Web browser 1 12 to the AOP 114 includes an overiay sources message 
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1 16 and a requested document message 1 1 7. while the AOP 1 1 4 provides the browser 112 with a merged document 
120. Data structures employed by the present invention, which can be stored in any form of electronic or magnetic 
memory, include an annotation directory 118 associated with the proxy server 1 30 and overlay groups 1 50, 1 52 asso- 
ciated with the Web servers 140 and 1 44. respectively. Each of the overlay groups 1 50, 1 52 can indtde multiple anno- 
5 tation overlays 151 a-d, 153e-f that include cross-references to documents such as the documents 146 (Doci) and 148 
(Doc2), which are stored on the Web servers 142 and 144. respectively. Where large numbers of annotation overlays 
are assodated with a particular overlay group, in the interests of scalability, that group can be split among several Web 
servers. The annotation directory 1 18 includes a plurality of annotation overlays 1 19a-f drawn from one or more of the 
overlay groups 150. 152. 

10 ' The AOP 1 14 is coupled to the Web servers 140-44 and communicates with the Web servers 140-44 using stand- 
ard internet (TCP/IP) and WWW (HTTP) protocols. The AOP 114 relays all commands from the Web browser 1 12 to 
the Web servers 1 40-44 and receives from the Web servers a copy of the requested document 1 15. Actions of the AOP 
1 14 are directed by local AOP procedures 1 1 4a and programs 1 13. 

Please note that the numbers of components shown in Figure 1 are merely exemplary. Also, the prefen-ed embod- 

15 iment depicted in Figure 1 is a generic configuration intended to illustrate the basic principles of the present invention. 
Consequently, the following descriptions of the prefened embodiment are applicable to other configurations of the ele- 
ments shown. Including a configuration where the annotation directory 118 and the annotation overlay proxy 114 are 
resident in the Web client 1 10. Having set out the elements of the present invention, these elements are now described 
in greater detail. The following descriptions, while general, will in places be directed to the situation where a user has 

20 issued a request to view the document 1 46. which is associated with the Web server 142. 

A. Web Browser: 

The Web browser 1 12. which displays the merged document 120. is functionally identical to prior art Web browsers. 

25 Thus, as set out in the background section, a user of the Web browser 112 accesses adocument 146 or 148 stored on 
the Web by first selecting a hypertext link (i.e.. a highlighted word or phrase) within a document currently being dis- 
played by the Web browser 1 1 2. Alternatively, a user can issue a document request by entering the desired document's 
URL in the Web browser. Similarly to existing browsers, the Web browser 112 acts on a user's document request by 
issuing a HTTP document request message 1 1 7 specifying the URL of the requested document Unlike existing brows- 

30 ers. the browser 112 issues the document request message 1 1 7 to the AOP 114. rather than to a firewall proxy or to a 
group of Web servers. However, in all respects, the HTTP document request message 1 17 is identical to one issued by 
existing browsers. 

The Web browser 1 12 also issues an HTTP sources message 1 16 to the AOP 1 14. This is a new message 0.e.. a 
message not currently used by existing Web browsers) that specifies the URLs of the oveday groups containing infbr- 

35 mation to be merged by the AOP 1 1 4 with the document identified in the document request message 1 1 7. For example, 
in Figure 1 , the sources message 1 1 7 indicates that the user wants to view annotation overlays from overlay groups 1 50 
and 152. corresponding to Sources 1 and 2, respectively This message can be issued by the Web browser 1 12 at any 
time after the AOP 114 has been initialized. 

The Web browser 1 12 can initialize the AOP 114 in two ways. Rrst. the user can enter the various oveday groMps 

40 they wish to view on a conunand line from the browser 1 1 2 or even when tiiey start the browser 112. The browser then 
initializes tfie AOP 1 14 and immediately issues a con-esponding sources message 1 16. causing the AOP 114 to build 
the annotation directory 1 18. Alternatively, a list of sources can be submitted to the browser 1 12 using a common gate- 
way interface (CGQ. after which the browser initiafizes the AOP 1 14 and issues the appropriate sources message 1 16 
to the AOP 114. 

45 

B. Annotat'on Overlay Processor (AQP): 

The AOP 1 14. like any other proxy (e.g, a firewall proxy), communicates with entities connected to the Web. such 
as ihe Web client 1 10 or the Web servers 140-42. using the standard HTTP proxy sender communications protocol. All 
so functions of the AOP 1 1 4 are directed by a set of local AOP procedures 1 1 4a. Like a firewall proxy, much of the AOP's 
job involves merely passing along messages between tiie Web browser 1 12 and the Web servers 140-44. For example, 
upon receiving a HTTP document request message 1 17 from the Web browser 1 12. the AOP 1 14 simply relays tiiat 
message to the Web servers 140-44. the appropriate one of which returns ttie requested document 

However, in contrast to a firewall proxy, which acts only as a kind of glass wall between ttie Web dient 1 1 2 and serv- 
55 ers 140-144. the AOP 1 14 can also transform tfie data being returned by ttie Web servers 1 40-144 to ttie Web browser 
1 12. In the preferred embodiment illustrated in Rgure 1 , tiiis trahstormation involves the AOP 1 1 4 merging a requested 
document returned by one of the Web servers 140-44 with annotation overiays 119 from ttie annotation directory 118 
that are associated with the requested document 

The AOP 114 builds an annotation directory 1 18 by searching the Web servers 140-44 for all annotation overlays 



4 



EP0 762 297A2 

contributed by the sources, or overlay groups, designated by the Web browser 11 2 in an HTTP sources message 1 1 6. 
and then storing those annotations in the annotation directory 1 18 as annotation overlays 1 19. Thus, in Figure 1, the 
annotation directory 118 includes all annotations from Source 1 (overlay group 150) and Source 2 (overlay group 152). 
which were specified in the message 1 16. Generally, the AOP 114 builds the annotation directory 118 only upon receiv- 
5 ing the HTTP sources message 116. 

Each annotation overlay 1 1 9 has five fields: (1 ) document URL. (2) source. (3) pattern. (4) action and (5) arg (short 
fa argument), which respectively tell the AOP 114: 

(1) the URL of the document to which the annotated pertains; 
10 (2) which source contributed the annotation overlay; 

(3) what specific part (or pattern) of the returned document the overlay pertains to; 

(4) the action to take with respect to the pattern; and 

(5) any additional information to assodate in the merged document with the pattern. This additional information can 
Include text or graphics to be inserted in the merged document or a designation of a type" annotation, such as 

IS grammar error ("gr). spelling error Csp"). "agree", or "disagree". 

To promote eff id ent retrieval of overlays, the AOP 1 1 4 orders the annotation overlays 11 9 on the URL of the anno- 
tated documents, although any other ordering of the overlays 1 19 is possible. These fields will be described in greater 
depth below. 

2Q Upon receiving the Image of a requested document from a Web server responding to an HTTP document request 
message 1 1 7 issued by the browser 1 12. the AOP 1 14 first identifies the set of annotation overlays 1 19 In the annota- 
tion directory 1 18 that are associated with the requested document's URL For example, assuming that the document 
. requested and returned was the document 1 46. this set would be the annotation overlays 1 1 9a.b. e.f (FIG. 1 ). This task 
IS made particularly easy in the preferred embodiment, where all annotation overlays 1 19 are organized by document 

25 URL The AOP 1 14 then aeates the merged document 120 by transforming the returned image 1 15 of the requested 
document 146 according to the information from the annotations 1 19a.b,e.f. This transformation is effected by the AOP 
114 adding HTML-formatted content to the merged document 120 so that the annotations are seamlessly integrated 
with the requested document and viewable using existing Web browsers such as the browser 1 12. 

In the prefen-ed embodiment, AOPs can be serially connected. Such an arrangement is shown in Rgure 2. where 

30 two AOPs 180. 182 are connected In series, with the AOP 180 being linked to the Web browser 1 12 and the AOP 182 
to the Web senders 1 40-44. As in Rgure 1 . each AOP has access to an annotation directory. 1 1 8a and 1 1 8b containing 
annotations derived from sources specified by a user using the HTTP sources message 116. However, unlike the 
arrangement shown in Rgure 1 . the annotation directories 1 18a and 1 18b include annotation overlays only from Source 
1 and 2. respectively. The need to serially connect AOPs can arise in several situations, two of which are now 

35 described. 

In a first situation, the annotation overlays from Source 1 and 2 might be formatted differently; consequently, the 
AOPs 180, 182 are spedalized to read the annotation overlays from Sources 1 and 2. respectively When the user asks 
tor source Information from diverse sources, such as Sources 1 and 2, the Web browser 112 opens as many different 
types of AOP as necessary to handle the different annotations. The merged document 1 84 returned to the Web browser 

40 1 12 Isthen created In two Steps. Rrst the AOP 182 merges the annotations from Source 2 with the returned document 
115. then the AOP 180 merges tiie annotations from Source 1 with the intermediate document 1 ^5\ 

A second situation where two AOPs are serially connected is where a source of annotations, such as Source 1. 
largely provides commentary on annotations from another source, such as Source 2. This is possit>le in the prefenred 
emtxxliment as each annotation overiay has an assodated URL that corresponds to the unique reference numbers 

45 1 19a.b.e.f ; 1 51 a<J and 1 53e-f . When this situation arises, two AOPs are aeated so that the annotations from Source 
1 are merged only after all of the annotations from Source 2. 

C. Overiav Groups 

so The same type of overlay information provided in the annotation directory 118 is also provWed by the overiay 
groups 150. 152, which, as set out above, are the original sources for the annotation overiays copied by the AOP 114 
to the annotation directory 118. The only real difference between the overiay groups 150, 152 and the annotation direc- 
tory 118 is that the overiay groups contain annotations for a multiplicity of documents confributed by a single source 
rather than a collection of annotations from a set of designated sources (i.e.. the sources designated to the AOP 1 14 

55 using the sources message 1 1 6). Consequentiy. the overlay groups are organized differentiy from the annotation direc- 
tory. 

In the preferred embodiment each annotation overiay group 150. 152 corresponds to a single source O-e.. Source 
1 and Source 2. respectively) and is formatted as shown in Table 1. 
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TABLE 1 



document_url 1 
s pattern a, action a 

arg a 
pattern b, action b 
arg b 

10 . • • • 



document_url 2 

pattern c, action c 

arg a 
pattern d, action d 
20 — arg b 

document_url 3 

pattern e, action e 
25 arge 

document^url 4 

pattern e, action e 
30 ^rge 



In this representation, each documenLurl corresponds to a document URL in the annotation directory 1 1 8 or over- 
35 lay group 150. 1 52 (FIG. 1). The other information fields: pattem (3), action (4) and arg (5). are the same as for a record 
1 19, 151. 153 in the annotation directory 1 18 or overlay groups 150, 152. respectively. 

For example, ttie overlay group 150 (FIG. 1) includes four annotation overlays 151a<J provided by Source 1. Of 
these overlays, the first two. 151a-b. pertain to tiie document 146 (Doci) , so tiiey are included under a heading refer- 
encing the URL of Doci. Similarly, the second two overlays 151c-d. pertain to the document 148 (Doc2) so they refer- 
40 ence the URL of Doc2. The overlay group 152 is formatted similarly to the overiay group 150 and includes annotation 
overlays 153e-f provided by Source 2 that are associated only with tfie document 146 (Doci). 

In tiie preferred embodiment, the overlay groups 150. 152 are written in HTML At all times document URLs are 
maintained in alphabetically sorted order. For example, some appropriate HTML for the overiay document of Table 1 is 
shown in Table 2. 

45 



SO 



55 
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TABLE 2 

<UL> 

^ <LI> document_url 1 

<DL> 

<DT> pattern a, action a 
<DD> arg a 
'io <DT> pattern b, action b 

<DD> arg b 



IS 



2Q 



25 



30 



</DL> 

< LI > document_url 2 

<DL> 

<DT> pattern c, action c 

<DD> arg a 

<DT> pattern d, action d 

<DD> arg b 

« • • 

</DL> 

< LI > document_url 3 

<DL> 

<DT> pattern e, action e 

<DD> arge 



</DL> 
<LI> document url4 
<DL> 

<DT> pattern e, action e 
35 <DD> arge 

</DL> 

• • • 

40 </UL> 



In Table 2, the terms between the paired *C and "T symbols are standard HTML commands, or tags, which allow 
the AOP to parse annotation overlays for the various fields. The tags used In Table 2 have the following meanings: 

45 

begin unordered list; 
list item; 

end unordered list; 
begin definition list; 
definition list term; 

definition of prior definition list term; and 
end definition list 

Having described the structure of the annotation directory 1 18 and the overlay groups 150, 152. the fields of the 
55 annotation overlays, common to both the annotation directory and the overlay groups, are now described in detail. 

D. Annotation Overlavs: 

In the prefenred embodiment the AOP 1 14 is designed to read annotation overlays that are written in HTML, which 



<UL) 
<U> 
(AJL> 
<DL> 
50 (DT> 
<DD> 
</DL> 
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allows authors to create annotations using widely available HTML authoring tools. Alternatively, special annotation 
authoring tools could be provided. 

Typically, overlay authors will place annotation overlays on the Web according to one of many authoring models. 
For example, in a cooperative (coop) model, a group of individuals contrfoute annotation overlays directly to one or more 

5 overlay groups 150. 152 associated with a cooperative to which the authors belong. For example, members of different 
political parties might contribute overlays to the overlay groups 150 and 152, respectively. In a magazine model, paid 
authors submit annotations to a centralized editor who then edits and publishes the authors' annotation overlays in an 
overlay group(s) managed by the editor. In this magazine model, it is envisioned that publishers of overlay groups will 
sell subscriptions or. like commercial broadcasters, advertising, to cover the costs of publication. 

10 As described above and shown in Figure 1 . in addition to specifying the document being annotated and the source 
of the annotations, each annotation overlay 119. 151. 153 includes the following information fields that allow the AOP 
1 1 4 to perform the con-ect transfbnnation on the correct part of the requested document: 

- pattern (3): 
IS ' action (4): and 
• arg (5). 

Each of these fields is described below in reference to the exemplary annotation overlay shown in Table 3. which 
represents a member of an overlay group such as tiie group 150. Please note that field 2 from the annotation overlays 
20 _1 19 is missing because source (2) is synonymous with the overlay group identifier. E.g., the overlay group 150 contains 
only annotation overlays autiiored by Source 1. 

TABLE 3 

25 (1) http://infoxern.sh/hypertext/V\A/WV/DaemonAJser 

(3){4) Tiles can be real or synthesized^ [Insert after sentence] 

(5) Unfortunately, there is no way to tell the 

^ difference between synthesized and real files; this 

makes it extremely difficult to reliably cache . 
HTML documents using the CERN server. 

35 



In the example of Table 3. the identifier field (1) indicates the URL (or document URL) of one respective document 
to be annotated. The remaining fields (3)-(5) represent the pattern, action and arg for the same annotation overlay. We 
40 proceed to descrft>e fields (3). (4) and (5). 

1. Annotation Overlays - Pattern Field 

A pattern (3) is a pattern of words or pixels in the requested document that the AOP 1 1 4 must operate on. The pat- 
45 tern field is necessary as the document URL in an annotation overlay does not provide fine location within a document 
but merely a pointer to the document as a wtiole. In the prefened embodiment, the pattern syntax consists of a list of 
words or pixels making up the pattem set off by balanced single quotes. For example, in the illustration above, the pat- 
tern the AOP 114 ntust search for in the document Identified by the document URL http://info.cern.sh/hyper- 
text/WWW/Daemon/User is: Tiles can be real or synthesized'. The remaining fields (4) and (5) tell the AOP 1 14 what 
so actions to take with regard to occurrences in the corresponding document, 

2. Annotation Overlays > Action Field: 

The action field (4) defines the action that the AOP 1 1 4 must take when merging an annotation overlay at the spec- 
55 if ied pattern in the requested document. In the preferred embodiment, this action can be selected from one of four basic 
operations: 

Insert Insert the contents of the arg field into the requested document at a specified locati n relative to the 

pattern. 
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Delete Delete the specified pattern or a range of words/images surrounding the specified pattern.. 

Replace Replace the specified pattern or a designated part of the document including the specif ted pattern with 
the contents of the erg field. 

5 

Run_Program Execute the identified program, which corresponds to a routine 1 1 3 that is associated with the AOP 114, 

In addition to one of the four basic operations, the action field (4) also includes several additional parameters that 
specify how the action is to be implemented with respect to the pattern field (3). In the prefen-ed embodiment, all of the 
10 arguments composing the action field follow the pattern field (3) and are enclosed in balanced square brackets. The 
syntax of the action field (4) is set out in TABLE 4. 

TABLE 4 

operation 

[matchl insert | 
Imatchl delete | 
[match] replace { 
[match] runjDrogram 

match 

match decimal number 

insert 

insert where | 

delete 

delete from where to where 
replace 

replace where ] 
runjjrogram 

run_program programjd where | 

where 

before location | 
after location 
location 

document j 
section I 
paragraph { 
sentence | 

word decimal number j 



In TatDle 4, the lx)lded terms are keywords that are used in a particular instruction. The unbolded terms represent 
variables or parts of an instruction. Optioneil parts of the action field, such as 'match' are shown enclosed in square 
brackets. For example, in the exemplary annotation overlay in Table 3, the entire action is: (Insert after sentence]. In tills 
example, the operation is Insert", the where part of the instruction is "after* and the location is "sentence*. The vertical 
50 bar (T) stands for "OR"; e.g. the location field can have one value selected from document, OR section OR para- 
graph, etc. 

The various parts of action field are defined in Table 5. 
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TABLE 5 



operation 



match 
wtiere 

document 



section 
paragraph 

sentence 

word decimal number 



As mentioned above, an operation Is the action to be implemented with respect to the 
pattern and is selected from one of insert, delete, replace and run_program. Insert, and 
replace operations take a where argument, which indicates to the AOP 114 the specific 
part to be operated on of a document including the pattern. Delete operations take an 
argument of: from where to where. The run_program operation takes two arguments, 
program Jd. which is the name of the program (local to the AOP 1 14) to be executed and 
%vhere". which identifies the specific part of the document to be operated on. 

A value that indicates which of multiple occunrences of the pattern the annotation over- 
lay applies to. 

As mentioned above, the where argument identifies a specific part of a document includ- 
ing tiie pattern. The where argument has two variations, before location and after loca- 
tion. In these variations location" is an enumerated variable that has five values: 
document, section, paragraph, sentence and word, which are described below. 

refers to tiie document containing the pattern. If preceded by before, the AOP 1 1 4 per- 
forms the operation at the beginning of the document. If preceded by after, the opera- 
tion is performed at the end of the document. For example, if the annotation overlay 
action field were: Tinsert before document!", tiie AOP 1 14 would perfonn the insert oper- 
ation at tiie t)eginning of the designated document 

refers to the section (designated using HTTP section tags) tfiat Includes the pattern. 

refers to tiie paragraph (designated using HTTP paragraph tags) that includes the pat- 
tem. 

refers to tiie sentence (determined by the AOP 114 parsing the requested document) 
tiiat includes the pattern. 

refers to a specific word (the one identified by the value of the "decima!_number" varia- 
ble) of tiie document containing tiie pattern. 



a. Annotation Overlays - Argument Reld: 

The argument field provWes the contents to be inserted by ttie AOP 1 14 in tfie requested document at tiie position 
identified by tiie where argument to tfie Insert and replace operations. For example, tiie exemplary annotation overlay 
of Table 3 Instructs tiie AOP 11 5 to Insert the phrase: 

Unfortunately, there is no way to tell the difference between synthesized and real files: this makes it extremely 
difficult to reliably cache HTML documents using the CERN server 
after the sentence that includes ttie pattern: 
Tiles can t>e real or synthesized'. 
Using this action f iekJ syntax, almost any type annotation operation can be specified for execution by an annotation 
overlay proxy such as ttie AOP 114. 



E. Method of the Prefered Embodiment 

Refen-ing to Rgure 3. there is shown a flow chart of tiie operation of the preferred embodiment The steps 
desaibed below as being executed by ttie AOP 1 1 4 are executed under ttie direction of the local AOP procedures 1 1 4a. 

As a first step, tfie user indicates to the browser 112 using an input device tfie overlay groups they wish to track 
(212). In response, the browser 1 12 determines whetfier it has already caused ttie annotation proxy server 130 to open 
an annotation overlay proxy tfiat is compatible witti tiie user-designated overlay groups (i.e.. an AOP ttiat knows how to 
merge annotations from ttie designated overlay groups witfi requested documents) (214). If such an AOP has not been 
opened (214-N). the Web browser 112 issues an open^AOP message to AOP server 130. which opens tfie correct type 
of AOP (216): tfie browser tfien Issues a sources message 1 16 to tfie newly available AOP (216). If ttie browser had 
previously caused the AOP sender to open a compatible AOP (214-Y), ttie browser determines whetfier ttie annotation 
directory 1 18 indudes overiays for tfie overlay groups specified by tfie user (218): if not (218-N), tiie browser issues an 
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HTTP sources message 11 6 to the previously-opened AOP 1 14 specifying the overlay groups not currently represented 

In the annotation directory 1 1 8 (220). 

As described above in reference to Figure 1. in response to the HTTP sources message, the AOP 1 14 accesses 

over the Web the overlay groups whose URLs match the source URLs in the sources message 116. The AOP 114 then 
5 copies all annotations from the designated overlay groups to the annotation directory 1 18 and orders all of the entries 

1 19 in the annotation directory 1 18 by document URL (222). At this point the AOP is initialized and waits for document 

request messages issued by the browser 1 1 2 (224). 

Whenever the user requests a document URL while using the Web browser, the browser issues a document 

request message 117 to the proxy server 130 (226). which the AOP 114 passes on to the Web server storing the 
w requested document. In response, that Web server returns the requested document to the AOP server 130. Once the 

requested document is returned to the proxy server 130 (228-Y), the AOP 114 creates a merged document 120 that 

represents a merger of the requested document and all annotation overlays from the annotation directory 1 18 that are 

associated with the requested document (230). The AOP 114 then returns the merged document 120 to the Web 

browser (232) for viewing. 

IS When aeating the merged document 120. the AOP 114 first copies the requested document to the merged docu- 
ment. The AOP 114 then adds the associated annotation overlays 1 19 to the merged document 120 in an order deter- 
mined by the precedence of the operation associated with each respective annotation overlay 119. In the preferred 
embodiment, the operations* precedence order, from highest to lowest, is: insert, replace, delete and program. This 
precedence order is inversely related to the degree of disruption caused in the merged document 120 by a particular 

£Q operation. For example, a delete operation from a source 2 (overlay group 1 52) overlay might delete the pattem needed 
for an insert operation from a source 1 (overlay group 150) overlay, but not vice versa. Any other precedence scheme 
could also be implemented. Of course, even given operator precedence. It is inevitable that sometimes the pattem 
required by an overlay is not in the merged document. When this is the case, the AOP 1 14 appends the annotation over- 
lay 1 19 including that pattern to the merged document and links that overlay to an "unassodated^annotation" icon dis- 

25 played at the beginning of the document By selecting an unassociated_annotation icon, a user may read the 
corresponding annotation overlay 1 1 9 which is displayed by the Web browser 112. 

When merging an overlay 119 specifying an insert operation, the AOP 1 14 appends the information from tiie arg 
field to the merge document locates designated occurrences of the pattern in the document (the designated occur- 
rences could be every occurrence or just those occurrences specified in the optional match field), then determines the 

30 position In the document relative to the designated occurrences where tiie information in the arg field is to be inserted. 
At that position, the AOP 114 adds an HTML tag{s) (or some other hyperlink indicator) to call out the annotation and 
links tfie appended information from the arg f iekJ to that tag(s). The AOP 1 1 4 also inserts HTML tags in the merged doc- 
ument around the pattern to convert the pattern into a hyperlink cue tied to tiie appended information from tiie arg f iekJ. 
When displaying the merged document the Web browser 1 12 can display an icon at the insertion point which a user 

35 can select to display the inserted text, or can directf y display the inserted text at the insertion point. In either situation, 
by selecting ttie displayed pattern, the Icon or the inserted text, tfie user can obtain Information 1 1 2 about the source of 
the annotation overlay Alternatively, tfie AOP 1 14 simply inserts the information from tfie arg field into the merged doc- 
ument at tiie location defined in the action field and adds HTML tags to the inserted text ttiat will cause the browser 
1 1 2 to highlight the inserted text when displayed. 

40 For example, Referring to Figure 4. tiiere is shown a nrierged document 320 resulting from ttie AOP 114 merging 
the annotation overlay 319a from ttie annotation directory 118 with tiie requested document 315 according ttie first 
insertion method described above. The annotation overlay 319a is derived from the one shown in Table 3; tiial is: 

(1) http://info.cern.sh/hypertext/WWW/Daemon/User 
45 (3)(4) 'Files can be real or synthesized' (Insert after sentence] 

(5) Unfortunately, there is no way to tell ttie difference between syntfiesized and real ffles; tiiis makes it extremely 
difficult to reliably cache HTML documents using the CERN server. 

As shown in Figure 4. the AOP 1 14 adds ttie inserted text from tfie field (5) to tiie top of ttie merged file 320, 
50 appends to ttie inserted text a source Wentif ier indicating ttiat the text came from Source 1 . and associates witii tfie text 
and source information a HTML cross reference begin and end tags 321a. 321b r(CR=insertir) designating tiie 
inserted text as "insertl". The AOP 114 then adds HTML begin and end annotation tags 323a. 323b (Mink to 
CR=insert1 T) to tfie merged file around all occun-ences of tiie pattem, "fOes can be real or syntfiesized". which occurs 
at two locations in tfie merged document. These tags reference tfie inserted text designated as ^nsertr and signal the 
55 Web browser to highlight tfie pattern when displaying tfie requested document. Because tfie operation associated witfi 
the annotation is Insert after sentenced tfie AOP 114 also adds an HTML tag 327 ("(include CR «= inserti 7) to tfie 
merged document 320 at the end of each sentence including the pattern. The tag 327 defines a hypertext link to tfie 
Inserted text associated with ttie reference Insertr. This merged document 320 can be displayed in ttie browser 112 
in any number of ways selected by tiie user of ttie browser. For example, tfie linked text might be displayed inline, or 
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linked to an icon, displayed at the position of the HTML tag 327. 

For example. Figure 5 shows how the merged file 320 might look when the overlay 319a is displayed as an inline 
annotation. Note that the Web browser 112 highlights the inserted text to alert the user. 

When the operation is replace, the AOP 114 locates designated occurrences of the pattern in the document then 
5 determines the part of the document containing the designated occurrences of the pattern which is to be replaced by 
the phrase or image in the arg field of the respective annotation overlay. The AOP 114 then performs the required 
replacement operation in the merged document 120 and places HTML around the replacement text that links the 
replacement text to an annotation specifying the source of the overlay, which the AOP 1 1 4 appends to the merged doc- 
ument 120. When a user selects the highlighted replacement text displayed by the Web browser 1 12, the browser 112 
to displays the source of the replacement text (e.g.. "Source 1*). 

When the operation is delete, the AOP 1 1 4 simply deletes the part of the merged document identified by the "from 
where to where** part of the delete annotation overlay. 

When the operation is run_program. the AOP 1 14 executes on the designated part of the requested document the 
local filter program identified by the programjd argument from the action field of the respective annotation overlay. For 
IS example, referring to Figure 1. the program 113 to be executed by the AOP 114 might be: translate_pgm_a or 
translalejpgm^b. These programs might perform operations such as summarizing, translating or decrypting the 
merged document 120. 320. 

While the AOP 114 could process annotation overlays in any order, in the preferred embodiment the AOP 114 
processes annotation overlays in an order determined by the relative precedence of the operation specified in the action 
20 JjiM of each respective annotation overiay. I.e., insert operations are always performed first then replace, delete and 
runjsrogram operations. 

F AltPrnfltivi^ gmbodiments 

25 A first alternative embodiment is shown in Rgure 6. In this embodiment, the AOP 1 14 does not prepare an anno- 
tation directory 1 18 in advance of a Web browser request for a document stored on a Web server. Instead, the AOP 114 
retrieves from the various overlay servers identified by the user network (with the sources message 1 1 7) the annotation 
overlays only for a particular document or documents and only upon receiving the aforementioned document request 
from the Web browser. The AOP 1 14 then temporarily stores the set off retrieved annotation overlays in memory, struc- 

30 tured similarly to the annotation directory 118 (Rg. 1). The AOP 1 14 then merges the annotations with the requested 
document exactly as described above. The advantage of this alternative embodiment is that the AOP 114 is required to 
store only a small set of all of the annotation overlays stored in the prefen-ed embodiment, which allows a user to more 
easily change the sources whose comments they wish to view. 

In another preferred embodiment, which is shown in Rgure 7. each annotation overlay, e.g.. the annotation overlay 

35 451 . can refer to a rangefeet of documents, rather than just a single document. This type of annotation overlay is useful 
where a source of commentary provides global comments on an entire dass of documents; e.g.. the entire Ixxly of work 
of an author or any publication of a particular company. These annotations are structured similarly to the style of anno- 
tation employed in the preferred embodiment (FIG. 1. Table 1). but can specify plural machine URLs (designating the 
address of a Web server) and plural document URLs (associated with each sender designated by a machine URL) for 

40 each pattern, action and arg triplet Rgure 7 also depicts another form off annotation 153h. which can be used in the 
present invention. This type of annotation 153h do^s not name a specific document but provides a search string: e.g., 
"All docs wHh acc =string1 & title including string2". that the AOP 114 matches to particular requested documents 
before nr^erging the accompanying pattern, action and arg triplet 

While the present invention has been desaibed with reference to a few specific embodiments, the description is 

45 illustrative of the invention and is not to be construed as limiting the invention. Various modifications may occur to those 
skilled in the art without departing from the true spirit and scope of the invention as defined by the appended claims. 

Ciaims 

50 1 . In a computer network incorporating a plurality of servers used to store documents, each said document having a 
unique document identifier, and a client computer having a browser configured to request and receive said docu- 
ments over said networK a system for providing annotation overlays for a requested document, said system com- 
prising: 

55 at least one stored overlay group associated with one of said servers, said overlay group encapsulating anno- 

tation overlays regarding at least one of said documents, each said stored overlay group having a unique 
source identifier; and 

an annotation overlay proxy configured to fomi a merged document by merging said requested document from 
a first server with associated annotation overlays regarding said requested document from specified overiay 
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groups and to relay said merged document to a receiver selected from another proxy or said browser 

The system of claim 1. wherein at least one overlay group encapsulates a plurality of distinct annotation overlays 
regarding one of said documents. 

The system of claim 1. wherein at least a subset of said annotation overlays each have associated therewith data 
indicating with which document or documents said each annotation overlay is to be associated. 

The system of claim 1 . further comprising: 

an annotation directory associated with said annotation overlay proxy, wherein said annotation directory stores 
sakj annotation overlays from said specified overlay groups, and wherein said annotation overlay proxy is con- 
figured, in response to a document request issued by said browser, to retrieve a set of associated annotation 
overlays related to said requested document from said annotation directory prior to merging said set of asso- 
ciated annotations and said requested document. 

The system of claim 4. wherein said annotation directory is created by said annotation oveday proxy, which, upon 
receiving a fist of said specified overlay groups identified by their unique source identifiers, is configured to retrieve 
all of said annotation overlays from said specified overlay groups and store said retrieved annotation overiays in an 
electronic memory coupled to said annotation overlay proxy, said stored annotation overlays forming said annota- 
tion directory 

The system of claim 1 . wherein said annotation overlay comprises: 

a document id specifying a set of documents to which said annotation overlay is applicable: 
a pattern in said set of documents; 

an action code specifying an action to be tal^n by said annotation overlay proxy with regard to said pattern in 
said set of documents when aeating said merged document: and 

an argument supplying additional infonnation to assist said annotation overlay proxy in executing said action. 

In a computer network incorporating a plurality of servers used to store documents, each said document having a 
unique document identifier, and a client computer having a t)rowser configured to request and receive said docu- 
ments over said network, a method for providing annotation overiays for a requested document, said method com- 
prising the steps of: 

associating at least one stored overlay group with one of said senders, said overlay group encapsulating anno- 
tation overlays regarding at least one of said documents, each said stored overlay group having a unique 
source identifier; 

forming a merged document by merging said requested document from a first server with associated annota- 
tion overlays regarding said requested document from spedfied overlay groups: and 
relaying said merged document to a receiver selected from another proxy or said browser. 

The method of claim 7, wherein said annotation overiay coniprises: 

a document kJ specifying a set of documents to which said annotation overiay is applicable; 
a pattern in said set of documents; 

an action code specifying an action to be taken with respect to said set of documents when forming sakJ 
merged document; and 

an argument supplying additional Information for said action code. 

The method of claim 8. wherein said action code for a subset of said annotation overlays is selected from the group 
consisting of a replacement code, an insertion code and a deletion code, such that: 

1) when said action code is said replacement code and said requested document is within said set of docu- 
ments, said action comprises substituting in said merged document said additional information for parts of said 
requested document having a specified relationship to said pattern; 

2) when said action code is said insertion code and said requested document is within said set of documents, 
said action comprises inserting in said merged document said additional information at a position set wherein 
each position in saki position set has a specified relationsNp to said pattern; and 
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3) when said action code is said deletion code and said requested document is within said set of documents, 
said action comprises preventing from appearing in said merged document any occurrence of a part of said 
requested document having a specified relationship to said pattern. 

5 10. The method of claim 8. wherein, when said action code is a program code and said requested document is within 
said set of documents, said action comprises executing a specified filter program on parts of said requested docu- 
ment having a specified relationship to said pattern so that corresponding parts of said merged document include 
a transformation of said parts operated on by said filter program. 

10 ' 11. A computer-readable memory that can be used to direct a computer to merge stored annotation overlays with doc- 
uments stored on a computer network to which said computer is coupled, said computer-readable memory com- 
prising: 

(1) an annotation overlay including: 

15 

(a) a document id specifying a set of documents to which said annotation overlay is applicable: 

(b) a pattern In said set of documents; 

(c) an action code specifying an action to be taken by said computer with regard to said pattern in said set 
of documents when creating said merged document; and 

20 _ (d) an argument supplying additional information to assist said annotation overlay proxy in executing said 

action with regard to said pattern; and 

(2) proxy procedures for forming a merged document by merging said annotation overlay with a requested doc- 
ument within said set of documents, said merging including performing said action specified by said action 

25 code. 
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ROSCHEISEN M ET AL: "BEYOND BROWSING: 
SHARED COMMENTS. SOAPS, TRAILS. AND • 
ON-LINE COMMUNITIES* 

* page 741. right-hand column, line 24 - 
page 747, left-hand column, line 19; 
figures 1-3.2.3 * 

COMPUTER NETWORKS AND ISDN SYSTEMS, 
vol. 27, no. 6, 1 April 1995. 
pages 911-918. XPO0Q498O89 
LALIBERTE 0 ET AL: "A PROTOCOL FOR 
SCALABLE GROUP AND PUBLIC ANNOTATIONS" 

* page 913, right-hand column, line 11 - 
page 916, left-hand column, line 26; 
figure 1 * 

IBM TECHNICAL DISCLOSURE BULLETIN, 
vol. 38, no. 9. 1 Septeiriier 1995. 
page 189 XP0O054e234 "CREATING 
DISTRIBUTED DOCUMENT FOLDERS IN A 
COLLABORATIVE VIEWING SESSION" 

* the whole document * 
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X : fuHmMy cdcvant if tmkca aloac 

Y : pafticuUrty relevant if combiaod with anothtf 

4ocancat of the saiac catcfOfy 
A : lechnological hackcraand 
O : aooowrittcn 4isdosiirc 
P : ialcfiBcdiatc 4ocunicnt 



T : chaffy or prmdplc utdtiiymg, the io««ntioo 
C : earlier patent document, but poMishcd oa, or 

after the filiag 4atc 
O : 4oauacnt dtcd in the ap pUatioa 
L : ^ocmacat dtetf for other rcasoas 

A : fBcmher of the saxac patent fzmtly, corrcspan^iae 
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